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Abstract. This paper will explore and evaluate the practical implications of 
six chosen visualization techniques for communicating uncertainty and 
provide suitability measures and guidelines for the practical use. Addition- 
ally, itwill detail ahuman subjects experiment design usi ng these visual iza- 
ti on techni ques for eval uati ng decisi on-maki ng i n bushf i re si tuati ons. 
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1. Introduction 

Making safety- critical decisions in potentially life-threatening hazardous 
situations is difficult, due to the time pressures involved with such a deci- 
sion. Additionally, the uncertain nature of the relevant information availa- 
bleaddsto the complexity. In Australia, there is a policy where it isthe per- 
sonal choi ce of the i ndi vi dual to vacate thei r home i n the i nstance of a bush- 
fire, rather than through the issuance of compulsory evacuation orders by 
the government. Thus, it is important to communicate predicted bushfire 
likelihood to a wide audience, from emergency response professionals to 
the general public, to aid in the decision of whether to stay or leave. This 
work is concerned with applying several different visualization techniques 
and evaluating them for decisi on- making in a bushfire setting. To achieve 
this, we utilize PHOENIX Rapidfire Bushfire Modelling software (which 



predicts fire spread and outputs burn likelihood prediction areas) and ex- 
amine different visualization techniques and their suitability for represent- 
ing the uncertainty associated with this data. This research relies on a 
methodology for an objective human subjects experiment, presenting nov- 
ice users with different scenarios and focusing on the decisions made from 
these different visual i zati ons. 

Whilst there is past experimental research into the effects of uncertainty 
visualization for decision- making in the context of natural hazards, these 
efforts have largely concentrated on experts and novices in situations such 
as avalanches (Kunz, Gret-Regamey, & H urni, 2011), tsunamis, floods (Trau 
& Hurni, 2007), seismic hazards and hurricanes (Pang, 2008). Past exper- 
i mental research has not focused on testi ng the effects of uncertai nty visual- 
ization from a bushfire decision- making situation. This research evaluates 
six carefully selected visualization techniques with human subjects in a 
bushfire context. The scenario used for testing these visualization tech- 
niques is one that is commonly encountered in bushfire situations- wheth- 
er the information that you have been presented with influences a decision 
of whether you "stay" or "leave" your place of residence. Thus, we will pre- 
sent users with scenarios where their house is marked on the display; to- 
gether with the burn likelihood represented using one of the above tech- 
niques. 

The work presented here aims to provide answers to the challenges and 
questions outlined above and detail the process taken to solve these chal- 
lenges. It outlines the decisions and rationalefor static technique selection, 
the use of the underlying PHOENIX Rapidfi re model (Figurella) to create 
the burn probability surfaces and lastly outlines some of the problems and 
choices made when designing the experiment interface. ESRI ArcGIS soft- 
ware (Figure llf), the Processing software package (Figure llh), PHP 
(Figure 12c), Structured Query Language (SQL) (Figure 12d) and J avaS- 
cript (Figure 12b) were the technologies chosen for the implementation of 
this experiment. 

This paper outlines the journey taken to design this experiment, from the 
selection of the representation techniques through to the design of the sce- 
narios and experiment interface itself. For a start, visualizations have to be 
carefully considered for their inclusion both from a suitability perspective 
as well as having past experience of success, as supported by literature. Af- 
ter the selection process is complete, the experimental design process be- 
gins. There are many important choices to be made, what technology do we 
choose to build the experiment and how can we maximize our success 
through the experiment design and the interface itself? 
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Figure L Schematic demonstrating interconnectedness of all the elements and 
processes used to design and create the experi ment interface. At a broad level there 
are two main elements of the design, the image maker (1) and delivery mechanism 
(2). The image maker (1) creates the scenarios that are delivered to the participant 
through the delivery mechanism (2) of which the main feature is the experiment 
i nterface ( 2a) I i nked to a supporti ng database ( 2d) . 



The primary focus/ aim of this research is to develop an experiment that is: 
intuitive, reusable, modifiable, robust, stand alone, with clean and unclut- 
tered vi sual i zati ons and i nterf ace. 



2. Choosing visualizations 

I n choosi ng whi ch vi sual izati on techni ques to be i ncl uded i n the experi ment 
we needed to achieve a fine balance between scenarios that have had past 
experience of success with those that had not been empirically tested be- 
fore, but had potential for success. Based on a literature review, six static 
techniques were selected for their inclusion in the experi ment - color (Fig- 
ure 2a), the PHOENIX representation (Figure 2f), FSPro representation 
(Figure 2b), transparency (Figure 2c), texture (Figure 2e) and a textual 
description (Figure2d). These will soon be explained in further detail. 

PHOENIX (Figure 2f) and FSPro (Figure 2b) were chosen as these repre- 
sentati ons are currently i n use i n the f i re servi ces sector today. These repre- 
sentations to our knowledge have not been previously tested for thei r suita- 
bility. The former is the visualization used to represent the current output 
from the PHOENIX model 1 , which isrepresented asa hard line showing the 
highest burn likelihood, followed by a dashed line for the lowest burn likeli- 
hood. The latter is the representation used by the FSPro Prediction Model 2 
which consists of bands of color to represent burn likelihood, with no clear 
value message. In addition, we chose three additional visualization tech- 
niques that are commonly used for communicating uncertainty, and are 
reported to be suited to representing quantitative categorical information 
(Bisantz et al., 2011; Bisantz et al., 2009; Leitner & Buttenfield, 2000). 
These techni ques- transparency (Figure 2c), texture (Figure 2e) and color 
value (Figure2a) - have not been previously evaluated for decision- making 
in bushfire situations. Color was chosen (with a scale of richer to weaker to 
represent decreasing uncertainty) because it is a variation to the FSPro sce- 
nario but with the addition of containing a clear value message. Leitner and 
Buttenfield (2000) found it to be successful for depicting uncertainty in a 
regional planning scenario and best results were observed where a richer 
color was used to represent more certain information and a weaker color 



1 Currently used by the Department of Sustainability and Environment and the Country Fire 
Authority in Victoria, Australia (http://www.bushfirecrc.com/news/news-item/mapping-ushfire- 
otential) 

2 Developed and used by the United States Department of Agriculture Forest Service in their 
Fire Decision Support Systems (http://wfdss.usgs.gov/wfdss_help/WFDSSHelp_FSPro_Ref.html) 



used to represent less certain information. Transparency was chosen as it 
proved successful for representing uncertainty in an experiment Bisantz et 
al. conducted using a military decision- making task in 2011 This work was 
the extension of a previous study on transparency using human subjects 
conducted by Bisantz et al. (2009) which found it to besuitablefor ranking 
levels of uncertai nty in thunderstorm mapping. 

Texture was described by Leitner and Buttenfield (2000) in their regional 
planning study, to be the next most successful following color saturation. 
Their representation of texture used finer texture to represent greater cer- 
tainty. 




Figure 2. Example of uncertainty visualizations chosen and their depiction in the 
experiment (clockwise from top left) (a) color, (b) FSPro representation, (c) trans- 
parency, (d) textual representation, (e) texture and (f) PHOENIX representation. 



To provide contrast, we also included a text- based representation of the 
predicted bushfire likelihood. This way we could evaluate whether repre- 
senting uncertainty through visualizations is actually more effective for de- 
cision-making than simple text based descriptions. 



3. Creating scenarios 

Having selected our candidates for inclusion in the experiment, we come to 
the process of creati ng the scenari os that opti mal I y di spl ay these candi dates 
so they can be tested. Additionally, there were decisions to be made around 
which software we used in this development process. PHOENIX Rapidfire 
is fire spread modelling software developed by Kevin Tolhurst and Derek 
Chong at the University of Melbourne. Currently, it is used by the Depart- 
ment of Sustainability and Country Fire Authority to generate fire spread 
prediction in an Australian landscape and is built as an interface which in- 
teracts with ESRI ArcGIS. Its inputs include weather, topography, fuel 
loads, fire history, roads and suppression (Paterson & Chong, 2011). 
PHOENIX Rapidfire fit our software requirements perfectly, as it is reliably 
used by fire authorities in Victoria today and can be used to depict burn 
likelihood asa probability surface. Wedid not want to build our own model 
or ti nker with existi ng less i deal models as the purpose of this work is not to 
create a model but rather to evaluate visualizations. We are trying to find 
the best method for representi ng the output of such models. 

3.1. Generating a surface 

The next step was to generate an uncertainty surface we could apply the 
above chosen visualization candidates to. I n order to create a surface with 
PHOENIX, ignition points must be placed. We decided on 15 different sce- 
narios so we could variably apply each of the candidates to these 15 scenari- 
os. The choice of 15 was influenced by the decision that we wanted each 
participant to view roughly 100 scenarios, and 15 scenarios x 6 representa- 
tions gives us 90 in total. We randomly placed 9 ignition points to generate 
each scenario. The 9 ignition points were created by placing a point in the 
centre of each grid cell of a randomly placed 3x3 grid. The random place- 
ment of this grid we restricted to a large area within the north-west of the 
study area. We did this to ensure the scenarios we generated did not fall 
outsi de the area of the map. 

Next we used the Grid Analysis simulation type in PHOENIX to batch pro 
cess the scenari os with a lm x lm resolution and a 24-hour prediction win- 
dow. The underlying weather conditions used for generation of the images 



was taken from an area in Victoria that was particularly adversely affected 
by the Black Saturday bushfi res of February 2009. 

3.2. Assigning burn likelihood 

Now we have a surface, but how do we assign values to create a probability 
surface? PHOENIX generates as part of its output a times impacted' col- 
umn, indicating the number of times a particular grid cell has been impact- 
ed by both fire and embers. The distribution of these was firstly examined 
and then probabilities assigned (Table 1). Percentages were chosen as the 
method of communication, as the literature in this field indicates that per- 
centages are most commonly used to represent probability for visualiza- 
tions (Bisantz, et al., 2011) . 

3.3. Creating the final image 

To create the overall image that is viewed by the participants in the experi- 
ment, unique house locations (Figure lie) were used and the true ignition 
point (Figure lib) was randomly selected from the nine used to create the 
scenarios. The houses were placed as a randomly stratified sample, however 
we evenly distributed the houses throughout each of the probability zones. 
The true ignition point we used to define the actual area burnt. When a par- 
ticipant chooses to stay or leave, this true ignition point is used to deter- 
mi ne whether they have made a ri ght or wrong choi ce. 

Together with the house location (Figurelle) and the surface (Figure 11c), 
the overall image included a hillshade image (Figure lid) provided by the 
Department of Sustainability and Environment as a backdrop. We chose to 
use solely a hillshade image because we didn't want prior knowledge of the 
area to influence decision- making or proximity of the houses to roads to 
affect the choi ces made i n the experi ment. H owever, we sti 1 1 wanted a back- 
drop that represented real world features. 



Times Impacted 


Burn Likelihood (%) 


8-9 


>80-100 


6-7 


>60-80 


4-5 


>40-60 


2-3 


>20-40 


1 


>0-20 



Table L Burn likelihood assigned to the surface according to the times impacted 
column in the output from PHOENIX, which is the number of times a particular 
grid cell has been impacted by fireor embers. 



4. Designing an experiment interface 

"Design is really an act of communication, which means having a deep un- 
derstanding of the person with whom the designer is communicating" 
(Norman, 1990). The interface is the vehicle that deli vers the scenarios that 
are viewed in the experiment and the only opportunity for the experi ment- 
ers to communicate with the experi mentees. This experiment is aimed at 
novices and hence it was designed that way. Regardless of all of the choices 
that were made previ ously, successful del i very of the experi ment is not pos- 
sible without a well-designed and executed experi ment interface. 

In designing our experiment interface, we reflect back on the overarching 
ai ms of this research - to desi gn an experi ment with an easy i ntuiti ve i nter- 
facethat doesn't require significant additional instruction and that would 
stand on its own. We wanted the display to be clear, uncluttered and as 
basic as possibleso the visualizations themselves could shine through. The 
experi ment system should be easily modifiable so it can be reused for future 
experiments of this nature. Another important requirement was a program 
to create the images themselves, as to create these manually would be an 
arduous process. 

To best achieve our goals, it was decided that two separate interfaces should 
be bui 1 1. One for automati c generati on of the vi sul izati ons from the separate 
shapefiles and hillshade image i.e. the image maker (Figure 11), and an- 
other for di spl ayi ng to the subj ects i .e. the del i very mecha ni sm ( F i gur e 12) . 
From the above requi rements the experi ment had to stand alone, so it could 
be run from any computer with a standard operating system and internet 
browser, it was clear that we needed a web interface and a database. PHP 
(Figure 12c) and J avaScript (Figurel2b) were chosen to build the web in- 
terface, a MySQL database (Figure 12d) was chosen for data storage. These 
were chosen for simplicity because the designers had prior knowledge and 
experience with these languages. Processing (Figure llh) was chosen to 
create the i mages as the authors have a strong desi re for usi ng open source 
software in keeping with sharing principles. Additionally, Processing han- 
dles the shapefiles created by the PHOENIX package well without any need 
for further manipulation or tinkering. 



4.1. The image maker 

Having chosen the Processing open source software package to create our 
images, we also discovered we needed to use an additional library - 
MapThing. The MapThing processing library was developed byj on Reades 
at University College London's Centre for Advanced Spatial Analysis 



(Reades, 2012). MapThing cleverly enables Processing to read in and han- 
dle ESRI compliant shapefiles and display them as part of a sketch. The 
visual display of the shapefiles can then be programmatically manipulated 
using processing. Using the MapThing library and Processing, a java pro- 
gram was written to import and display the burn likelihood surfaces (Fig- 
ure 11c), raster hillshade backdrop (Figure lid) and 5400 unique house 
locations (Figurelle). The program then automatically generates images 
based upon a combination of one of the 5 different visualizations (Figure 
llg) and unique house locations overlaid on the hillshade background 
(Figure 3). It also automatically generates the legend, shown to the right of 
the image, and matches the legend to the visualization of choice. These im- 
ages are automatically batch saved as portable network graphic (PNG) im- 
ages (Figure lli). 




Figure 3. Example of the output generated from the Processi ng program. 



4.2. The delivery mechanism 

The next step in the experiment design is the creation of the delivery mech- 
anism (Figure 12) encompassing the web interface (Figure 12a) and linked 
database (Figure 12d). 



I n order to do this we needed first to take a brief look at the setup of our 
experiment. The experiment is a human subjects experiment for 60 partici- 
pants where each participant views 90 scenarios. A 'turn up' fee is paid to 
parti ci pants for their parti ci pati on i n the experi ment. They wi 1 1 be gi ven 30 
seconds to view each scenario and must make a decision of whether to stay 
or leave and press the corresponding button. The time limit of 30 seconds 
was chosen as we did not want to rush the participants into a decision but 
did want to impose reasonable time limit bounds around the experiment. 
The chosen time limit was based on initial pilot experiments; we initially 
started with 15 seconds but this was deemed by pilot participants to be too 
short. We also included a questionnaire at the end to facilitate further anal- 
ysis of the decisions made. This questionnaire included some basic demo- 
graphic questions about the participant as well as some questions to judge 
their maths ability, map reading ability and experience with bushfires. 

Now reflecting back on the design process itself. We wanted the image con- 
taining the visualization (Figure 3) to have maxi mum screen real estate, so 
this was placed in the centre of the interface and was made as large as pos- 
sible (Figure 12a). As the experiments were timed, we obviously needed a 
timer. We decided on a count down timer, so participants could easily see 
how long they had remaining. In keeping with a clean interface, we placed it 
in the top left corner close to the image so it could be seen with a quick 
gl ance wi thout the parti ci pant havi ng to I ook away from the i mage. The stay 
and go buttons we placed at the top of the screen close to each other but not 
so close that the wrong button could be accidentally pressed (Figurel2a). 
We didn't want people to have to move their mouse around too much. The 
design of the database was also based around simplicity and guided by the 
design of the experiment interface. We had a number of elements that 
needed to be stored; the ordering of the experiments for each individual, 
image names, burnt or not burnt information, time taken, response and all 
the information captured through the questionnaire. 

All of this led to a database build that consisted of three tables: experiment 
data, user data and questionnaire (Figure 4). The experiment data table 
stores information about the ordering of the experiment, including unique 
experiment numbers, image names and burnt and not burnt information. 
The user data table stores information about the actions of each user 
throughout the experi ment i ncl udi ng ti me taken, response and whether the 
response was correct or incorrect. The questionnaire table stores basic de- 
mographic information about the user as well as some basic questions 
about the experiment. All the images are stored as PNG images in the data- 
base and the ordering is contained in the experiment table. A unique, non- 
consecutive six-digit experiment number was created as an identifier for 



each participant so the anonymity of the participants is maintained at all 
times. 
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Figure 4. Database schema showi ng tables created for the experi ment database. 

I n keeping with an experiment design that is fully stand-alone, the experi- 
ment was administered using a URL that the participants entered into a 
web browser. The six-digit number was entered into the experiment screen 
and this loaded the instructions and the experiment. Responses made by 
each participant were recorded in the user data and questionnaire tables 
correspondi ng to the unique parti ci pant number. 



5. Conclusion 

In conclusion, this paper outlines the journey we took for designing an ex- 
peri ment for evaluating uncertai nty for decision- making in a bushfi re situa- 
tion. Our journey begins at the selection of visualizations then we travel 



through to the creation of scenarios and culminates at the design of the ex- 
perimental interface. 

We now address each of the mai n ai ms we had for the i nterface i n turn and 
outline how these have been met through the design process. The portabil- 
ity of the interface was adhered to as we created a web-based interface ad- 
ministered through a URL that works with all firefox browsers. The instruc- 
tions for the experiment are in-built with the URL and the responses are 
recorded in a MySQL database. Initial pilot sessions using these experi- 
ments have proven successful and participants have commented through 
the questionnaire that the experiment interface was intuitive and easy to 
use with one participant commenting "very simple interface, clear instruc- 
tions". The experiment is built with reusability in mind as the images and 
orderi ng are stored as separate components withi n the database. Therefore, 
if we wanted to use a different set of i mages or change/ adj ust the orderi ng, 
it would be a simple process of loading fresh images into the database (Fig- 
ure lli) and/ or making changes to the experiment data database table 
(Figure lid). Furthermore, as each of the components of the experiment 
design are distinct from each other, it would be easy to reuse this experi- 
ment for a different hazard application area (eg. by substituting a flood 
model in place of PHOENIX (Figure 11a)). 

I niti al pi I ot sessi ons usi ng these experi ments have proved successful . 
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